Automating Reinforcement Learning with Example-based Resets

Deep reinforcement learning has enabled robots to learn motor skills from environmental interactions with minimal to no prior knowledge. However, existing reinforcement learning algorithms assume an episodic setting, in which the agent resets to a fixed initial state distribution at the end of each episode, to successfully train the agents from repeated trials. Such reset mechanism, while trivial for simulated tasks, can be challenging to provide for real-world robotics tasks. Resets in robotic systems often require extensive human supervision and task-specific workarounds, which contradicts the goal of autonomous robot learning. In this paper, we propose an extension to conventional reinforcement learning towards greater autonomy by introducing an additional agent that learns to reset in a self-supervised manner. The reset agent preemptively triggers a reset to prevent manual resets and implicitly imposes a curriculum for the forward agent. We apply our method to learn from scratch on a suite of simulated and real-world continuous control tasks and demonstrate that the reset agent successfully learns to reduce manual resets whilst also allowing the forward policy to improve gradually over time.Comment: 8 pages, 6 figures; accepted for publication in the IEEE Robotics and Automation Letters (RA-L); source code available at https://github.com/jigangkim/autoreset_rl ; supplementary video available at https://youtu.be/himd0Z5b64

General Chemical Reaction Network Theory for Olfactory Sensing Based on G-Protein-Coupled Receptors : Elucidation of Odorant Mixture Effects and Agonist-Synergist Threshold

This work presents a general chemical reaction network theory for olfactory sensing processes that employ G-protein-coupled receptors as olfactory receptors (ORs). The theory is applicable to general mixtures of odorants and an arbitrary number of ORs. Reactions of ORs with G-proteins, both in the presence and the absence of odorants, are explicitly considered. A unique feature of the theory is the definition of an odor activity vector consisting of strengths of odorant-induced signals from ORs relative to those due to background G-protein activity in the absence of odorants. It is demonstrated that each component of the odor activity defined this way reduces to a Michaelis-Menten form capable of accounting for cooperation or competition effects between different odorants. The main features of the theory are illustrated for a two-odorant mixture. Known and potential mixture effects, such as suppression, shadowing, inhibition, and synergy are quantitatively described. Effects of relative values of rate constants, basal activity, and G-protein concentration are also demonstrated

Quasi-coherent fluctuation measurement with the upgraded microwave imaging reflectometer in KSTAR

The microwave imaging reflectometer (MIR) is the leading diagnostic tool for study of density fluctuations in KSTAR. For last three years since 2014, major components such as the multi-frequency probe beam source, multi-channel detector array, signal processing electronic system, data acquisition system, and optical system have been gradually upgraded. In this paper, the detailed system upgrade with test results in the laboratory and/or plasma is given, and analysis results of a distinctive fluctuation structure referred to as the quasi-coherent mode (QCM) measured by the upgraded MIR system for an L-mode discharge are presented. Cross-coherence analysis with multiple channels shows that the QCM is localized in a core region and appears to be driven by electron temperature gradient for the discharg

Comparison of The Genome Profiles Between Head and Body Lice

The body louse (Pediculus humanus humanus) is known to have diverged from the head louse (P. humanus capitis) but genomic differences between these two subspecies still remain unexplored. To compare genomic profiles between head and body lice, whole genome sequences of head lice were determined by next generation sequencing methods based on both Illumina Genome analyzer and Roche GS FLX pyrosequencing and compared with the reference genome sequences of the body louse. Total consensuses generated by mapping to the body louse genome in conjunction with de novo assembly of head louse genome sequences revealed a head louse genome size of 110 Mbp with a 96% coverage of the body louse genome sequences. A total of 12,651 genes were predicted from the head louse genome sequences although more precise assembly and functional annotation of the genome is required for a more accurate gene count. Among the 873 genes that were putatively specific to the head louse, 15 genes were confirmed to be transcribed in both head and body lice, suggesting the previously estimated gene number of the body louse was likely underestimated. The single nucleotide polymorphism analysis showed that the nucleotide diversity of genome between head and body lice was 2.2%, which was larger than that of the transcriptome between head and body lice. An endosymbiont genome analysis showed that the composition of endosymbionts in head lice was similar to that of body lice and Candidatus Riesia pediculicola was the primary endosymbiont in both head and body lice

Enhanced cytotoxic effect of radiation and temozolomide in malignant glioma cells: targeting PI3K-AKT-mTOR signaling, HSP90 and histone deacetylases

BACKGROUND: Despite aggressive treatment with radiation therapy and concurrent adjuvant temozolomide (TMZ), glioblastoma multiform (GBM) still has a dismal prognosis. We aimed to identify strategies to improve the therapeutic outcome of combined radiotherapy and TMZ in GBM by targeting pro-survival signaling from the epidermal growth factor receptor (EGFR). METHODS: Glioma cell lines U251, T98G were used. Colony formation, DNA damage repair, mode of cell death, invasion, migration and vasculogenic mimicry as well as protein expression were determined. RESULTS: U251 cells showing a low level of methyl guanine transferase (MGMT) were highly responsive to the radiosensitizing effect of TMZ compared to T98G cells having a high level of MGMT. Treatment with a dual inhibitor of Class I PI3K/mTOR, PI103; a HSP90 inhibitor, 17-DMAG; or a HDAC inhibitor, LBH589, further increased the cytotoxic effect of radiation therapy plus TMZ in U251 cells than in T98G cells. However, treatment with a mTOR inhibitor, rapamycin, did not discernibly potentiate the radiosensitizing effect of TMZ in either cell line. The mechanism of enhanced radiosensitizing effects of TMZ was multifactorial, involving impaired DNA damage repair, induction of autophagy or apoptosis, and reversion of EMT (epithelial-mesenchymal transition). CONCLUSIONS: Our results suggest possible strategies for counteracting the pro-survival signaling from EGFR to improve the therapeutic outcome of combined radiotherapy and TMZ for high-grade gliomas

Prion disease induced alterations in gene expression in spleen and brain prior to clinical symptoms

Prion diseases are fatal neurodegenerative disorders that affect animals and humans. There is a need to gain understanding of prion disease pathogenesis and to develop diagnostic assays to detect prion diseases prior to the onset of clinical symptoms. The goal of this study was to identify genes that show altered expression early in the disease process in the spleen and brain of prion disease-infected mice. Using Affymetrix microarrays, we identified 67 genes that showed increased expression in the brains of prion disease-infected mice prior to the onset of clinical symptoms. These genes function in many cellular processes including immunity, the endosome/lysosome system, hormone activity, and the cytoskeleton. We confirmed a subset of these gene expression alterations using other methods and determined the time course in which these changes occur. We also identified 14 genes showing altered expression prior to the onset of clinical symptoms in spleens of prion disease infected mice. Interestingly, four genes, Atp1b1, Gh, Anp32a, and Grn, were altered at the very early time of 46 days post-infection. These gene expression alterations provide insights into the molecular mechanisms underlying prion disease pathogenesis and may serve as surrogate markers for the early detection and diagnosis of prion disease

Design and fabrication of adjustable red-green-blue LED light arrays for plant research

BACKGROUND: Although specific light attributes, such as color and fluence rate, influence plant growth and development, researchers generally cannot control the fine spectral conditions of artificial plant-growth environments. Plant growth chambers are typically outfitted with fluorescent and/or incandescent fixtures that provide a general spectrum that is accommodating to the human eye and not necessarily supportive to plant development. Many studies over the last several decades, primarily in Arabidopsis thaliana, have clearly shown that variation in light quantity, quality and photoperiod can be manipulated to affect growth and control developmental transitions. Light emitting diodes (LEDs) has been used for decades to test plant responses to narrow-bandwidth light. LEDs are particularly well suited for plant growth chambers, as they have an extraordinary life (about 100,000 hours), require little maintenance, and use negligible energy. These factors render LED-based light strategies particularly appropriate for space-biology as well as terrestrial applications. However, there is a need for a versatile and inexpensive LED array platform where individual wavebands can be specifically tuned to produce a series of light combinations consisting of various quantities and qualities of individual wavelengths. Two plans are presented in this report. RESULTS: In this technical report we describe the practical construction of tunable red-green-blue LED arrays to support research in plant growth and development. Two light fixture designs and corresponding circuitry are presented. The first is well suited for a laboratory environment for use in a finite area with small plants, such as Arabidopsis. The second is expandable and appropriate for growth chambers. The application of these arrays to early plant developmental studies has been validated with assays of hypocotyl growth inhibition/promotion and phototropic curvature in Arabidopsis seedlings. CONCLUSION: The presentation of these proven plans for LED array construction allows the teacher, researcher or electronics aficionado a means to inexpensively build efficient, adjustable lighting modules for plant research. These simple and effective designs permit the construction of useful tools by programs short on electronics expertise. These arrays represent a means to modulate precise quality and quantity in experimental settings to test the effect of specific light combinations in regulating plant growth, development and plant-product yield

Tetra­kis(1,2-dimethoxy­ethane-κ2 O,O′)ytterbium(II) bis­(μ2-phenyl­selenolato-κ2 Se:Se)bis­[bis­(phenyl­selenolato-κSe)mercurate(II)]

The title salt, [Yb(C4H10O2)4][Hg2(C6H5Se)6], consists of eight-coordinate homoleptic [Yb(DME)4]2+ dications (DME is 1,2-dimethoxy­ethane) countered with [Hg2(SePh)6]2− di­anions. The cations and anions have twofold rotation and inversion symmetry, respectively. The Yb centre displays a square-anti­prismatic coordination geometry and the Hg centre has a distorted tetra­hedral coordination environment. One phenyl­selenolate anion and one methyl group of a DME ligand are disordered over two positions with equal occupancies. This structure is unique in that it represents a less common mol­ecular lanthanide species in which the lanthanide ion is not directly bonded to an anionic ligand. There are no occurrences of the [Hg2(SePh)6]2− dianion in the Cambridge Structural Database (Version of November 2007), but there are similar oligomeric and polymeric Hgx(SePh)y species. The crystal structure is characterized by alternating layers of cations and anions stacked along the c axis

The Reproducibility of Indoor Air Pollution (IAP) Measurement: A Test Case for the Measurement of Key Air Pollutants from the Pan Frying of Fish Samples

To assess the robustness of various indoor air quality (IAQ) indices, we explored the possible role of reproducibility-induced variability in the measurements of different pollutants under similar sampling and emissions conditions. Polluted indoor conditions were generated by pan frying fish samples in a closed room. A total of 11 experiments were carried out to measure a list of key variables commonly used to represent indoor air pollution (IAP) indicators such as particulate matter (PM: PM1, PM2.5, PM10, and TSP) and a set of individual volatile organic compounds (VOCs) with some odor markers. The cooking activity conducted as part of our experiments was successful to consistently generate significant pollution levels (mean PM10: 7110 μg m−3 and mean total VOC (TVOC): 1400 μg m−3, resp.). Then, relative standard error (RSE) was computed to assess the reproducibility between different IAP paramters measured across the repeated experiments. If the results were evaluated by an arbitrary criterion of 10%, the patterns were divided into two data groups (e.g., <10% for benzene and some aldehydes and >10% for the remainders). Most noticeably, TVOC had the most repeatable results with a reproducibility (RSE) value of 3.2% (n=11)